A Hilbert-nagata Theorem in Noncommutative Invariant Theory

Nagata gave a fundamental sufficient condition on group actions on finitely generated commutative algebras for finite generation of the subalgebra of invariants. In this paper we consider groups acting on noncommutative algebras over a field of characteristic zero. We characterize all the T-ideals of the free associative algebra such that the algebra of invariants in the corresponding relatively free algebra is finitely generated for any group action from the class of Nagata. In particular, in the case of unitary algebras this condition is equivalent to the nilpotency of the algebra in Lie sense. As a consequence we extend the Hilbert-Nagata theorem on finite generation of the algebra of invariants to any finitely generated associative algebra which is Lie nilpotent. We also prove that the Hilbert series of the algebra of invariants of a group acting on a relatively free algebra with a non-matrix polynomial identity is rational, if the action satisfies the condition of Nagata.